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(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a three- 
dimensional image generating system which can generate 
a three-dimensional image stably even in the outdoors, 
and a three- dimensional information service system. 
SOLUTION: A three-dimensional shape can be 
measured with high accuracy using a plurality of (more 
than one) portable terminals with camera by preparing 
bases for installing these portable terminals with camera 
and calibrating them. An image can be picked up with a 
stabilized luminance even in the outdoors by a 3D 
imaging mode where flash light is used while setting 
predetermined shutter speed and iris or the shutter speed 
or iris is adjusted automatically through a unit for 
measuring illuminance. Furthermore, an arrangement for 
transferring three-dimensional data measured through 
use of the portable terminal with camera or three- 
dimensional data stored in a server to other terminal is 
realized. 
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(54) THREE-DIMENSIONAL IMAGE GENERATING SYSTEM, THREE- DIMENSIONAL IMAGE 
METHOD AND THREE- DIMENSIONAL INFORMATION SERVICE SYSTEM, AND PROGRAM 
MEDIUM 

(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a three-dimensional image 
generating system which can generate a three-dimensional image stably |: 
even in the outdoors, and a three- dimensional information service 
system. 

SOLUTION: A three-dimensional shape can be measured with high 
accuracy using a plurality of (more than one) portable terminals with 
camera by preparing bases for installing these portable terminals with 
camera and calibrating them. An image can be picked up with a 
stabilized luminance even in the outdoors by a 3D imaging mode where 
flash light is used while setting predetermined shutter speed and iris or 
the shutter speed or iris is adjusted automatically through a unit for 
measuring illuminance. Furthermore, an arrangement for transferring 
three-dimensional data measured through use of the portable terminal 
with camera or three-dimensional data stored in a server to other 
terminal is realized. 
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CLAIMS 



[Claim(s)] 

[Claim 1] Two or more cameras photoed from an eye direction which is the three-dimension image generative 
system which performs a three-dimension shape measurement or three-dimension image generation processing 
based on stereo mensuration, and is different in the image for measurement, It has the plinth which sets said 
two or more cameras as fixed physical relationship. At least one camera of two or more of said cameras To the 
server which performs a three-dimension shape measurement or three-dimension image generation processing 
The personal digital assistant camera which is a personal digital assistant camera which has the means of 
communications which can transmit photography image data, and has said means of communications The three- 
dimension image generative system characterized by having the configuration which transmits data required for 
the photography image, the three-dimension shape measurement, or three-dimension image generation 
processing photoed with said two or more cameras to said server. 

[Claim 2] Said three-dimension shape measurement or data required for three-dimension image generation 
processing is a three-dimension image generative system according to claim 1 characterized by including the 
parameter which shows the physical relationship of two or more of said cameras installed in said plinth. 
[Claim 3] Said server is the three-dimension image generative system according to claim 1 characterized by to 
be the configuration of extracting the parameter which shows the physical relationship of a camera based on 
the discernment data of the plinth received from the personal digital assistant camera which has the table 
which matched the discernment data of said plinth, and the parameter which shows the physical relationship of 
two or more cameras installed in this plinth, and has said means of communications, and performing a three- 
dimension shape measurement or three-dimension image generation processing. 

[Claim 4] Said plinth is a three-dimension image generative system according to claim 1 characterized by 
having the data signal way which outputs and inputs between cameras the control signal for the photography 
conditioning of two or more cameras installed in this plinth. 

[Claim 5] Said plinth is a three-dimension image generative system according to claim 1 characterized by 
having the data signal way which outputs and inputs between cameras the synchronizing signal of two or more 
cameras installed in this plinth. 

[Claim 6] Said server is a three-dimension image generative system according to claim 1 characterized by being 
the configuration of performing matching processing between the images photoed with two or more cameras 
based on the epipolar line field which set up line breadth. 

[Claim 7] Said server is a three-dimension image generative system according to claim 1 characterized by being 
the configuration of performing yes-or-no processing of access to the data which have a storage means to 
store the three-dimension shape-measurement data or the three-dimension image data based on the image 
received from said personal digital assistant camera, and were stored in said storage means based on the 
authentication processing between the terminals in which a server and a communication link are possible. 
[Claim 8] It is the three-dimension image generation method which performs a three-dimension shape 
measurement or three-dimension irnage generation processing based on stereo mensuration. The image 
photography step which installs two or more cameras in the plinth set as fixed physical relationship, and photos 
the image for measurement with two or more cameras from a different eye direction, At least one camera of 
two or more of said cameras is minded. Photography image data. The three-dimension image generation method 
characterized by having the data transmitting step which transmits a three-dimension shape measurement or 
data required for three-dimension image generation processing, and the data-processing step which performs a 
three-dimension shape measurement or three-dimension image generation processing based on the received 
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data in a server. 

[Claihi 9] Said three-dimension shape measurement or data required for three-dimension image generation 
processing is a three-dimension Image generation method according to claim 8 characterized by including the 
parameter which shows the physical relationship of two or more of said cameras installed in said plinth. 
[Claim 10] Said server is a three-dimension image generation method according to claim 8 characterized by 
extracting the parameter which shows the physical relationship of a camera based on the discernment data of 
the plinth received from the personal digital assistant camera which has the table which matched the 
discernment data of said plinth, and the parameter which shows the physical relationship of two or more 
cameras installed in this plinth, and has said means of communications, and performing a three-dimension shape 
measurement or three-dimension image generation processing. 

[Claim 11] The three-dimension image generation method according to claim 8 characterized by outputting and 
inputting between cameras the control signal for the photography conditioning of two or more cameras installed 
in a plinth through a plinth in said image photography step. 

[Claim 12] The three-dimension image generation method according to claim 8 characterized by outputting and 
inputting between cameras the synchronizing signal of two or more cameras installed in a plinth in said image 
photography step. 

[Claim 13] Said data-processing step is a three-dimension image generation method according to claim 8 
characterized by performing matching processing between the images photoed with two or more cameras based 
on the epipolar line field which set up line breadth. 

[Claim 14] Said three-dimension image generation method is a three-dimension image generation method 
according to claim 8 characterized by performing yes-or-no processing of access to the data stored in said 
storage means based on the authentication processing between the terminals in which a server and a 
communication link are still more possible. 

[Claim 15] Two or more cameras of the user possession which is the three-dimension information service 
system which offers a three-dimension shape measurement or three-dimension image generation processing 
service based on stereo mensuration, and photos the image for measurement from a different eye direction. It 
has the plinth of the user possession which sets said two or more cameras as fixed physical relationship. At 
least one camera of two or more of said cameras The personal digital assistant camera which is a personal 
digital assistant camera which has the means of communications which can transmit for photography image 
data to a user's contract server, and has said means of communications Data required for the photography 
image, the three-dimension shape measurement, or three-dimension image generation processing photoed with 
said two or more cameras are transmitted to said server. Said server The three-dimension information service 
system characterized by having the configuration which performs a three-dimension shape measurement or 
three-dimension image generation processing based on received data. 

[Claim 16] Said server is a three-dimension information service system according to claim 15 characterized by 
to be the configuration of performing yes-or-no processing of access to the data which have a storage means 
to store the three-dimension shape-measurement data or the three-dimension image data based on the image 
received from said personal digital assistant camera, and were stored in said storage means based on the 
authentication processing between the terminals in which a server and a communication link are possible. 
[Claim 17] It is the program offer medium which offers materially the computer program which makes a three- 
dimension shape measurement or three-dimension image generation processing perform on computer system 
based on stereo mensuration. The image photography step which said computer program installs two or more 
cameras in the plinth set as fixed physical relationship, and photos the image for measurement with two or 
more cameras from a different eye direction, At least one camera of two or more of said cameras is minded. 
Photography image data, The program offer medium characterized by having the data transmitting step which 
transmits a three-dimension shape measurement or data required for three-dimension image generation 
processing, and the data-processing step which performs a three-dimension shape measurement or three- 
dimension image generation processing based on the received data in a server. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the system, three-dimension image generation method, and 
three-dimension information service system which generate the so-called three-dimension image. Furthermore, 
it is related with the three-dimension image generative system, three-dimension image generation method, and 
three-dimension information service system which made possible 3D shape measurement stabilized also out in 
the fields by using flash plate photography, a camera plinth, etc. of a camera for a detail. 
[0002] 

[Description of the Prior Art] The technique of catching a three-dimension configuration using light, an image 
(photograph), etc. is divided roughly, and is divided into the active technique (Active vision) and the passive 
technique (Passive vision). As the active technique, (1) laser beam, a supersonic wave, etc. are emitted, the 
amount of reflected lights and time of concentration from an object are measured, and there is a method of 
making a contour line form by the Moire fringe, and acquiring three-dimension information etc. by the technique 
of extracting depth information, the method of presuming an object configuration from image information, such 
as geometric deformation of an object surface pattern, using the special pattern light sources, such as (2) slit 
light, and (3) optical processing. On the other hand as the passive technique, the knowledge about how an 
object appears, the light source, lighting, shadow information, etc. is used. The ocellus stereoscopic vision which 
presumes three-dimension information from the image of one sheet, and human being's eye are resembled. A 
dynamic-image scene to the 2 eye (or many eyes) stereoscopic vision which presumes the depth information 
on each pixel certainly by the triangulation principle, an optical flow, etc. are detected, and, generally the 
movement stereoscopic vision which presumes depth information from the place of the obtained optical flow is 
known. 

[0003] In order to measure three-dimension configurations, such as an industrial use product, with a sufficient 
precision until now, some three-dimension sensing equipments (or three-dimension digitizer), such as a 
practical instrumentation system which uses slit light and asks for a depth map on real time, and a 2 eye (or 
many eyes) stereo system by which the high precision calibration of the property between the interior of a 
camera and a camera was carried out, are announce. 

[0004] In order to measure the three-dimensions configuration on the front face of an object, based on a 
triangulation principle, a target configuration or depth is as measurable by the same principle, as human being 
senses a target configuration and depth by both eyes with matching of each pixel between images using the 
image observed with two or more cameras with which views differ. Generally this is called "stereo **" or 
"stereo three-dimension image measurement." 

[0005] The fundamental principle of stereo ** is triangulation and the most fundamental configuration is stereo 
** with two cameras. That is, the same object can be observed as an image with two cameras in a different 
view, and the target three-dimension coordinate location can be obtained by the difference in the projection 
location to the image top (generally a location gap of the projecting point between such images is called 
parallax), then, the thing for which the same object in a different projection location between images on either 
side is found — stereo ** — most importantly and generally it is called "corresponding^points attachment." 
[0006] On the other hand, matching of the projecting point location between images is not free at all, and as 
shown in drawing 1 . its corresponding points pi, or p2 or p3 exist in ml on a certain straight line Lp in the 
image of another side observed with another view, or m2 or m3 to the point mb on one image. Generally, this 
straight line is called an epipolar line (Epipolar Line). 
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[0007] It is divided roughly into three. Pixel-based matching, Area-based matching, and Feature-based 
matching, as the technique of "corresponding-points attachment" often used conventionally. [bibliography:(1) 
C.Lawrence Zitnick and Jon A.Webb which searches for correspondence of the point in one image as it is by 
the image of another side with Pixel-based matching : Multi-baseline Stereo Using Surface Extraction, 
Technical Report, CMU-CS -96-196,] (1996), With Area-based matching, correspondence of the point in one 
image [: [ bibliography ] (2) Okutomi who searches using the surrounding local image pattern of the point when 
searching by the image of another side, Stereo matching, Institute of Electronics. Information and 

Communication Engineers paper magazine D-II using two or more base lengths, Vol.J75-D-n, No.8, and 
pp.1 31 7-1 327 (1992), (3) Yokoyama, Miwa, Ashigahara, Koyaizu, a wood, after : Stereo Camera System and Its 
Application. SRF'97, (1997), and (4) Kimura: Video rate stereo machine. Robotics Society of Japan, Vol.13. 

No.3. and pp.322-326 (1995), (5) Development of the **♦*. ♦***, Kimura, Kawamura, Yoshida. and Ota:video rate 
stereo machine, Robotics Society of Japan, Vol.15, No.2. and pp.261 -267 (1997), (6) Yamaguchi. high ground, 
Inokuchi : Stereo matching for the stone image measurement using the adaptation window method. Human 
sciences, a computer. Vol.32. No.10. and pp.55-60 (1996). (7) Yokoya : The latest signal-processing synthesis 
special number The latest subject of a computer vision, a system / control / information, Vol.38, No.8, pp.436- 
441 (1994),]. With Feature-based matching The descriptions, such as a shade edge, are extracted from an 
image. Since it matches only using the description between images A certain [bibliography: (8) H.H.Baker and 
T.O.Binford : Depth from edge and intensity based stereo. In Proc.IJCAr81 (1981), (9) I Ishiyama, ****, Kawai. 

Tomita : The correspondence candidate retrieval in a segment based stereo, Shingaku Giho, Vol.96 and 
No. 136 (1997), (10) W.E.LGrimson : Computational experiments witha feature based stereo algorithm and IEEE 
Trans.PAMI, Vol.7, No.1. pp.1 7-34 (1985).]. 

[0008] How to ask for the corresponding points of stereo ** by general Area-based matching is explained. 
Changing a location on the epipolar line of the image Im2 (x y) which set the local window W as the surroundings 
of a certain observing point, and was observed with the camera of another side by making it into a template in 
the image Iml (x y) observed with the camera used as criteria, as shown in drawing 2 , it matches by being set 
- up retrieval within the limits, and asks for whenever [ coincidence ] by the bottom type. 
[0009] 
[Equation 1] 

[0010] In the above-mentioned formula, deltax and delta y are the movement magnitude of the location on an 
epipolar line. When R (x y) becomes min, whenever [ between windows / coincidence ] can be the highest, can 
make movement magnitude deltax and delta y of the template at that time the parallax of the observing point, 
and can compute the three-dimension configuration or depth of the observing point by the triangulation type. 
By repeating such matching processing, the three-dimension configuration data corresponding to all pixels can 
be obtained. 

[001 1] In fact, it is very difficult to ask for matching of all parts correctly on an image. Matching between 
images is because "ambiguity (Ambiguity)" essentially exists. Although template matching will be performed in a 
very small field and the high part of whenever [ coincidence ] will be most found also in the example of drawing 
2 if it sees from the whole image, possibility that a pattern which was alike with the scene is and of by the 
way existing is high. Moreover, even if it sees the same object in a scene, the pattern on a right-and-left image 
is not completely in agreement with the geometric distortion when seeing with a different view, the difference in 
the property of a camera, etc. 
[0012] 

[Problem(s) to be Solved by the Invention] Although it becomes possible to propose various technique, to 
reduce the ambiguity of matching of stereo and to decide correspondence to be accuracy more in order to 
solve these problems and to perform corresponding-points attachment between images to accuracy more, it 
cannot be said that it is actually enough. Since the image gray level by the illumination light cannot change a lot 
and cannot necessarily keep regularity exact for the physical relationship between cameras by change of a 
photography environment especially, the present condition is that there is actually no robust three-dimension 
shape-measurement method or robust system which can respond to ambient light, a delicate change of camera 
physical relationship, etc. out in the fields. 

[0013] Then, this invention aims at offering the robust three-dimension mensuration and the robust 
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instrumentation system which can respond to minute change of the stereo image acquiring method by the 
photography environment and camera physical relationship. The stereo system and three-dimension 
information service system whose 3D shape measurement stabilized also out in the fields becomes possible are 
realized by using flash plate photography, a camera plinth, etc. of a camera as a concrete application in the 
location using the personal digital assistant with a camera (two or more sets) which friends have. 
[0014] 

[Means for Solving the Problem] This invention Is made in consideration of the above-mentioned technical 
problem. The 1st side face Two or more cameras photoed from an eye direction which is the three-dimension 
image generative system which performs a three-dimension shape measurement or three-dimension image 
generation processing based on stereo mensuration, and is different in the image for measurement, It has the 
plinth which sets said two or more cameras as fixed physical relationship. At least one camera of two or more 
of said cameras To the server which performs a three-dimension shape measurement or three-dimension image 
generation processing The personal digital assistant camera which is a personal digital assistant camera which 
has the means of communications which can transmit photography image data, and has said means of 
communications It is in the three-dimension image generative system characterized by having the configuration 
which transmits data required for the photography image, the three-dimension shape measurement, or three- 
dimension image generation processing photoed with said two or more cameras to said server. 
[0015] Furthermore, the three-dimension Image generative system of this invention sets like 1 operative 
condition, and said three-dimension shape measurement or data required for three-dimension image generation 
processing is characterized by including the parameter which shows the physical relationship of two or more of 
said cameras installed in said plinth. 

[0016] Furthermore, it is characterized by to be the configuration of the three-dimension image generative 
system of this invention setting like 1 operative condition, extracting the parameter which shows the physical 
relationship of a camera based on the discernment data of the plinth received from the personal digital 
assistant camera which said server has the table which matched the discernment data of said plinth, and the 
parameter which shows the physical relationship of two or more cameras installed in this plinth, and has said 
means of communications, and performing a three-dimension shape measurement or three-dimension Image 
generation processing. 

[0017] Furthermore, the three-dimension image generative system of this invention sets like 1 operative 
condition, and said plinth is characterized by having the data signal way which outputs and inputs between 
cameras the control signal for the photography conditioning of two or more cameras installed in this plinth. 
[0018] Furthermore, the three-dimension image generative system of this invention sets like 1 operative 
condition, and said plinth is characterized by having the data signal way which outputs and inputs between 
cameras the synchronizing signal of two or more cameras installed in this plinth. 

[0019] Furthermore, the three-dimension image generative system of this invention sets like 1 operative 
condition, and said server is characterized by being the configuration of performing matching processing 
between the images photoed with two or more cameras based on the epipolar line field which set up line 
breadth. 

[0020] Furthermore, it has a storage means store the three-dimension shape-measurement data or the three- 
dimension image data based on the image which set like 1 operative condition and said server received from 
said personal digital assistant camera of the three-dimension image generative system of this Invention, and It 
is characterized by to be the configuration of performing yes-or-no processing of access to the data stored in 
said storage means based on the authentication processing between the terminals in which a server and a 
communication link are possible. 

[0021] Furthermore, the 2nd side face of this Invention is a three-dimension image generation method which 
performs a three-dimension shape measurement or three-dimension image generation processing based on 
stereo mensuration. The image photography step which installs two or more cameras in the plinth set as fixed 
physical relationship, and photos the image for measurement with two or more cameras from a different eye 
direction. At least one camera of two or more of said cameras is minded. Photography image data, The data 
transmitting step which transmits a three-dimension shape measurernent or data required for three-dimension 
image generation processing. It is In the three-dimension image generation method characterized by having the 
data-processing step which performs a three-dimension shape measurement or three-dimension image 
generation processing based on the received data in a server. 

[0022] Furthermore, the three-dimension image generation method of this invention sets like 1 operative 
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condition, and said three-dimension shape measurement or data required for three-dimension image generation 
processing is characterized by including the parameter which shows the physical relationship of two or more of 
said cameras installed in said plinth. 

[0023] Furthermore, it is characterized by for the three-dimension image generation method of this invention to 
set like 1 operative condition, to extract the parameter which shows the physical relationship of a camera 
based on the discernment data of the plinth received from the personal digital assistant camera which said 
server has the table which matched the discernment data of said plinth, and the parameter which shows the 
physical relationship of two or more cameras installed in this plinth, and has said means of communications, and 
to perform a three-dimension shape measurement or three-dimension image generation processing. 
[0024] Furthermore, it is characterized by outputting [ for the three-dimension image generation method of this 
invention to set like 1 operative condition, and ] and inputting between cameras the control signal for the 
photography conditioning of two or more cameras installed in a plinth through a plinth in said image photography 
step. 

[0025] Furthermore, it is characterized by for the three-dimension image generation method of this invention 
setting like 1 operative condition, and outputting and inputting between cameras the synchronizing signal of two 
or more cameras installed in a plinth in said image photography step. 

[0026] Furthermore, the three-dimension image generation method of this invention sets like 1 operative 
condition, and said data-processing step is characterized by performing matching processing between the 
images photoed with two or more cameras based on the epipolar line field which set up line breadth. 
[0027] Furthermore, it is characterized by for the three-dimension image generation method of this invention 
setting like 1 operative condition, and performing yes-or-no processing of access to the data stored in said 
storage means based on the authentication processing between the terminals in which a server and a 
communication link are possible. 

[0028] Furthermore, the 3rd side face of this invention is a three-dimension information service system which 
offers a three-dimension shape measurement or three-dimension image generation processing service based on 
stereo mensuration. Two or more cameras of the user possession which photos the image for measurement 
from a different eye direction, It has the plinth of the user possession which sets said two or more cameras as 
fixed physical relationship. At least one camera of two or more of said cameras The personal digital assistant 
camera which is a personal digital assistant camera which has the means of communications which can transmit 
for photography image data to a user's contract server, and has said means of communications Data required 
for the photography image, the three-dimension shape measurement, or three-dimension image generation 
processing photoed with said two or more cameras are transmitted to said server. Said server It is in the three- 
dimension information service system characterized by having the configuration which performs a three- 
dimension shape measurement or three-dimension image generation processing based on received data. 
[0029] Furthermore, it has a storage means store the three-dimension shape-measurement data or the three- 
dimension image data based on the image which set like 1 operative condition and said server received from 
said personal digital assistant camera of the three-dimension information service system of this invention, and 
it is characterized by to be the configuration of performing yes-oi^no processing of access to the data stored 
in said storage means based on the authentication processing between the terminals in which a server and a 
communication link are possible. 

[0030] Furthermore, the 4th side face of this invention is a program offer medium which offers materially the 
computer program which makes a three-dimension shape measurement or three-dimension image generation 
processing perform on computer system based on stereo mensuration. The image photography step which said 
computer program installs two or more cameras in the plinth set as fixed physical relationship, and photos the 
image for measurement with two or more cameras from a different eye direction, At least one camera of two or 
more of said cameras is minded. Photography image data, The data transmitting step which transmits a three- 
dimension shape measurement or data required for three-dimension image generation processing, It is in the 
program offer medium characterized by having the data-processing step which performs a three-dimension 
shape measurement or three-dimension image generation processing based on the received data in a server. 
[0031] The program offer medium concerning the 4th side face of this invention is a medium which offers a 
computer program in a computer-readable format to the general purpose computer system which can perform 
various program codes, for example. Especially the gestalten, such as transmission media, such as storages, 
such as CD, and FD. MO, or a network, are not limited for a medium. 

[0032] Such a program offer medium defines the collaboration-relation on the structure of the computer 
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program and offer medium for realizing the function of a computer program predetermined in a computer 
system top, or a function. If it puts in another way, by installing a computer program in computer system 
through this offer medium, on computer system, a collaboration-operation is demonstrated and the same 
operation effectiveness as other side faces of this invention can be acquired. 

[0033] The purpose, the description, and advantage of further others of this invention will become clear by 
detailed explanation based on the example and the drawing to attach of this invention mentioned later. 
[0034] 

[Embodiment of the Invention] In the three-dimension image generative system of this invention, and a three- 
dimension image generation method, the basal principle of the stereo method as a three-dimension image 
applied follows the stereo method of drawin g 1 explained previously. The configuration of drawin g 1 is explained 
briefly. Drawing 1 is the conceptual diagram of general camera arrangement of stereo **, and the depth 
measurement by the triangular principle, as point ** was carried out. When observing three points PI, P2, and 
P3 in three-dimensions space with a camera 1 (it is called a criteria camera), in an image side, it is projected on 
the same location mb, but if it observes with a camera 2 (it Is called a detection camera), in an image side, it will 
be projected on ml, m2, and m3. Then, if the die length of a camera lens and the physical relationship between 
cameras (it is called the base line) get to know beforehand and the location gap between mb, and m1, m2 and 
m3 (it is called parallax) is detectable, the distance (it is called depth) of triangular measurement principle 
machine ******. a camera, and P1, P2 and P3 is measurable. 

[0035] Drawing 2 shows the conceptual diagram of the technique of asking for the corresponding points by 
general template matching, in order to detect the location gap between mb, and m1, m2 and m3 (parallax). 
Changing a location on the epipolar line of the image Im2 which set the local window W as the surroundings of 
the observing point (x y) in the criteria image Iml, and was observed with the detection camera by making it 
into a template, it matches by being set-up retrieval within the limits, whenever [ coincidence ] makes the 
highest location the location of corresponding points, and parallax is searched for. However, it is very difficult to 
ask for matching of all parts correctly on an image in fact. Matching between images is because "ambiguity 
(Ambiguity)" essentially exists. Since the image gray level by the illumination light cannot change a lot and 
cannot necessarily keep regularity exact for the physical relationship between cameras by change of a 
photography environment especially, the present condition is that there is actually no robust three-dimension 
shape-measurement method or robust system which can respond to ambient light, a delicate change of camera 
physical relationship, etc. out in the fields. 

[0036] In this invention, the robust three-dimension mensuration and the robust instrumentation system which 
can respond to minute change of the stereo image acquiring method by the photography environment and 
camera physical relationship are proposed. It is [0037] which proposes the stereo system and three-dimension 
information service system whose 3D shape measurement stabilized also out in the fields becomes possible by 
using flash plate photography, a camera plinth, etc. of a camera in the location, using a personal digital assistant 
with a camera (personal digital assistant camera) as a concrete application. Drawing 3 is drawing explaining the 
example of an outline configuration of the three-dimension image generative system of this invention, a three- 
dimension image generation method, and a three-dimension information service system. 

[0038] In drawin g 3 , it is the conceptual diagram of the three-dimension shape-measurement system by the 
personal digital assistant with a camera (personal digital assistant camera), the personal digital assistant 
301,302 with two or more cameras is installed on the plinth which was alike beforehand and was prepared, it is 
the lighting environment, for example, the natural light in the outdoors, or the interior of a room and the usual 
lighting environment of arbitration, for example, a person's face is photoed. Although the three-dimension 
configuration of a face can be measured on that spot if a predetermined three-dimension configuration 
measurement program is applied using the calculation function when using a personal digital assistant with a 
calculation function, a three-dimension configuration cannot be measured in a personal digital assistant without 
a calculation function. 

[0039] Then, the photoed image of two or more sheets is transmitted to the decided server 303 which exists 
automatically from the personal digital assistant 301,302 with a camera. Automatic transmission is realized by 
considering the photography image photoed with each camera, and a parameter required for three-dimension 
configuration measurement as the configuration which makes a certain decided server perform a transmitting 
processing program by performing mode setting as 3D photography mode to the personal digital assistant 
301,302 with a camera. 

[0040] In a server 303 side, three-dimension shape-measurement processing is automatically performed using 
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the image sent from the personal digital assistant 301,302 with a camera, and the three-dimension configuration 
data of the obtained face are saved at the folder of the corresponding user attached to a server, or it returns 
to a user s personal digital assistant. Moreover, according to a user's password authentication etc., it supposes 
that it is accessible and the three-dimension image accumulated in the server 303 can consider three- 
dimension measurement data as the configuration transmitted to friends through a server 303. 
[0041] By the way, in order to measure the target three-dimension configuration with a sufficient precision, as 
previous drawing 1 and 2 explained, it is required to perform exact matching between images. Therefore, the 
robust stereo matching technique in which it can respond to minute change of the calibration of the physical 
relationship between the robust photography approach by the illumination light of (1) photography environment, 
and (2) camera parameter and a plinth and the physical relationship between (3) cameras is needed. 
[0042] Drawin g 4 shows drawing explaining two or more examples of the robust photography approach by the 
illumination light of some photography environments. Each example of drawin g 4 is set as the location as for 
which the criteria camera and the detection camera carried out predetermined distance alienation as two 
cameras. Each camera location is determined as the distance defined beforehand and the distant location by 
the plinth 40 for gestalt terminal installation. 

[0043] Drawing 4 (a) is the configuration of forming the illuminance measuring instrument 41 1 in the criteria 
camera 41, measuring the illuminance of ambient light, such as sunlight and lighting, with the illuminance 
measuring instrument 41 1, and setting up automatically the shutter speed of the criteria camera 41 and the 
detection camera 42 with the value. That is, the configuration of drawing 4 (a) is extracted and is the shutter 
speed priority method of immobilization. In addition, an illuminance measuring instrument is good also as the 
configuration which prepares in the criteria camera 41 or the detection camera 42, and sets up both shutter 
speed, or a configuration which forms the illuminance measuring instrument which became independent to each 
of the criteria camera 41 and the detection camera 42, and sets up shutter speed uniquely, respectively. 
[0044] Drawing 4 (b) measures the illuminance of ambient light with the illuminance measuring instrument 431 in 
the criteria camera 43, and shows the block diagram which sets up automatically the drawing value of the 
criteria camera 43 and the detection camera 44 with the value. That is, it is the drawing priority method of 
shutter speed immobilization. In addition, an illuminance measuring instrument is good as the configuration 
which prepares in the criteria camera 43 or the detection camera 44, and sets up both drawing values, or a 
configuration which forms the illuminance measuring instrument which became independent to each of the 
criteria camera 43 and the detection camera 44. and sets up a diaphragm value uniquely, respectively as well as 
(a). 

[0045] In order to remove the effect of ambient light, drawing 4 (c) forms a stroboscope 451 in the criteria 
camera 45, fixes the shutter speed of the criteria camera 45 and the detection camera 46, and a diaphragm, and 
shows the block diagram which photos a clear face image by stroboscope light. In addition, like (a) and (b), even 
if it prepares a stroboscope in the criteria camera 45 or the detection camera 46, it is good also as a 
configuration which prepares the stroboscope which became independent to each of the criteria camera 45 and 
the detection camera 46. 

[0046] In addition, a criteria camera and a detection camera all need to synchronize and it is necessary to 
operate by the above-mentioned photography method. A synchronizing signal is outputted and inputted through 
the plinth which sets up a criteria camera and its camera. Although the latter part also as a personal digital 
assistant and another object explains a plinth, it considers as the configuration which folded up to one personal 
digital assistant and was attached as a configuration or a cash-drawer configuration. Various control signals, 
such as above-mentioned shutter speed, a diaphragm, a stroboscope, and a synchronizing signal, are considered 
as the configuration which makes a plinth and a personal digital assistant connect, output and input by the 
connection connector. 

[0047] The example of the plinth configuration of a camera and another form is shown in drawin g 5 . Drawin g 5 
shows the conceptual diagram of the plinth applied to stereo ** which uses two sets ( drawin g 5 (a)) or three 
cameras ( drawing 5 (b)). In drawin g 5 (a) or drawin g 5 (b), a personal digital assistant 1,501,503 is used as 
criteria cameras, and a personal digital assistant 2,502,504 and a personal digital assistant 3,505 are used as 
detection cameras. On the plinth, a location and an include angle are fixed, and each camera carries out 
predetermined distance alienation of the distance between cameras, and is enabling photography of the 
photographic subject in the specific direction. 

[0048] The distance and the relative angles of rotation alpha and beta between the cameras installed in the 
plinth are decided with the measurement precision required as the lens die length of a camera, the distance for 
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measurement (face), a measurement range and CCD size, and the number of pixels. For example, lens die- 
length =8mm [ of a camera ], distance =600mm [ for measurement (face) ], measurement range =300mm, and 
CCD size =1/3" (4.8mmx3.6mm), the number of CCD pixels = when it considers as 640x480 and the 
measurement precision of about *^1 .0mm, the angle of rotation alpha with a distance relative at about 1 20mm 
between plinths should just set up with about 8.3 degrees. Moreover, the distance and the relative angles of 
rotation alpha and beta between exact plinths are decided by the calibration using a standard known image 
pattern. 

[0049] Thus, a plinth becomes it is possible to set the location of each camera and a direction as a specific 
position, and possible [ determining a parameter required for three-dimension configuration measurement based 
on the setting camera location of a plinth ]. Therefore, in the configuration of above-mentioned drawin g 3 , 
when the personal digital assistant equipped with the camera transmits the discernment data of a plinth to a 
server 303 with a photography image, based on reception discernment data, a correspondence parameter is 
extracted from the table which matched the plinth discernment data with which the server 303 is beforehand 
registered into the server, and the parameter data corresponding to each plinth, and activation of the image 
processing which applied the parameter to a photography image is attained. With this configuration, from a 
personal digital assistant, a parameter cannot be transmitted but measurement of a three-dimension 
configuration and generation processing of a three-dimension image can be performed only by transmitting the 
discernment data of a plinth at a server side. In addition, it becomes possible to extract the parameter 
corresponding to each camera registered into the server side, and to perform processing by transmitting the 
discernment data of a camera to a server, also with each parameter of the camera itself. 
[0050] Drawing 6 shows the known image pattern used and a parameter list to ask, in order to search for the 
physical relationship between a camera parameter and the plinth for two or more set camera setup. The 
technique of a calibration is indicated by JP,1 1-53548,A, JP,1 1-53549,A, etc. Drawing 6 (a), (b), and (c) are the 
known checker pattern images observed with the criteria camera in distance 1 (near distance), distance 2 
(middle distance of a measurement range), and distance 3 (far distance), respectively. 

[0051] In addition, the checker pattern image of three sheets in three distance can be similarly observed with a 
detection camera. And the interior parameter of a camera (focal length, distortioncoefficient, distortion center, 
aspect ratio) and the parameter (rotation angles, translational components) showing the physical relationship 
between plinths (that is, physical relationship between a criteria camera and a detection camera) can be 
decided by the double lump by the criteria image and reference image which were observed in each distance. 
[0052] However, neither by the machine precision of a plinth, nor dispersion of the body size of a camera, since 
the physical relationship between a criteria camera and a detection camera changes somewhat, the 
corresponding points of an observing point (x y) necessarily exist on the epipolar line called for by the 
calibration at the time of shipment of a camera. 

[0053] So, in consideration of elements, such as machine precision of a plinth, and dispersion of a camera, the 
epipolar line previously explained using drawing 2 is set up as an epipolar line (field) with line breadth, and this 
invention is searched for corresponding points based on an epipolar line with a field. 

[0054] Drawin g 7 shows the conceptual diagram of the stereo matching method by the retrieval on an epipolar 
line (field) with line breadth. By the conventional matching method, in order to find the corresponding points (x', 
y') on a detection image to the observing point (x y) in the criteria image Iml, template matching is performed 
by making it into a template using the window W centering on an observing point (x y). moving on the epipolar 
line of a detection image, and let the point (x*. y') that a matching score serves as max be corresponding points. 

[0055] The line breadth of an epipolar line is decided after taking into consideration elements, such as machine 
precision of a plinth, and dispersion of a camera. Template matching was performed using the window (template) 
centering on the observing point (x y) in the criteria image Iml, moving on the epipolar line field of a detection 
image, and the point (x', y') that a matching score served as max was made into corresponding points. A highly 
precise three-dimension configuration can be searched for by such matching. 

[0056] The line breadth of an epipolar line sets up the field R of ** (x y) as follows based on the epipolar line L 
(x y) called for by the calibration at the time of shipment. Here, it is assumed that a criteria camera and a 
detection camera meet horizontally and are installed. 
[0057] 
[Equation 2] 



http:/ /www4.ipdl.ncipi.go.jp/ cgi-bin/tran,web_cgi_ejje 



2005/02/15 




8/1 1 



K 



R(x,y)- \jL(x,y,) 



[0058] However, after K takes into consideration elements, such as machine precision of a plinth, and dispersion 
of a camera, it is decided experientially. Template matching was performed using the window (template) 
centering on the observing point (x y) in the criteria image Iml, moving on the epipolar line field of a detection 
image, and the point (x', y') that a matching score served as max was made into corresponding points. A highly 
precise three-dimension configuration can be searched for by such matching. 

[0059] Drawing 8 is a processing flow which shows the three-dimension shape-measurement procedure of using 
a personal digital assistant with a camera. After photoing the face image as a three-dimension shape- 
measurement object, an image, the camera used at the time of photography, the parameter of a plinth or a 
camera, and the discernment data of a plinth are automatically transmitted to a certain decided server. And 
with the three-dimension shape-measurement processing software (for example, the stereo matching technique 
and software) installed on the server, it asks for the three-dimension configuration data of a face, and saves on 
a server. Moreover, the three-dimension configuration data stored in the server can be processed and 
transmitted by the specific user according to authentication of a password etc. Therefore, it becomes possible 
to transmit and receive the image which each photoed through a server through the personal digital assistant 
with a camera which friends have, respectively. It is possible to, perform various processings, such as three- 
dimension configuration measurement and three-dimension image generation, on that spot in a personal digital 
assistant with a calculation function on the other hand, without through a server. 

[0060] The processing flow of drawing 8 is explained. At step S801, the personal digital assistant 1 with a 
camera and the personal digital assistant 2 with a camera are installed in a plinth. At steps S802 and S803, the 
image of an object, for example, a face, is photoed in the personal digital assistants 1 and 2 with a camera, 
respectively. At this time, the mode is set as 3D mode. By setting it as 3D measurement mode, as drawin g 4 
explained, it has equipment which measures an illuminance by diaphragm as fixed shutter speed, using flash 
plate light, and is set as the photography mode in which a diaphragm is automatically adjusted with fixed shutter 
speed. Furthermore, in a personal digital assistant with a camera without a calculation function, processing 
which transmits a photography image to the server set up beforehand by setup in 3D measurement mode is 
performed. 

[0061] At step 8804,805, it transmits to the server set up beforehand, the processing, i.e., the photography 
image, as 3D mode processing. In addition, this processing is not performed when processing within the end of a 
local. When performing an image transfer, it transmits also about the parameter or discernment data of a plinth. 
The parameters of a plinth are the distance between cameras and the relative angles of rotation alpha and beta 
of a plinth, as drawing 5 explained. As mentioned above, these are decided with the measurement precision 
required as the lens die length of a camera, the distance for measurement (face), a measurement range and 
CCD size, and the number of pixels, and are called for beforehand. 

[0062] In a server, based on the image data and parameter which were received from each camera, it asks for 
the three-dimension configuration data of a face for photography, and saves with the three-dimension shape- 
measurement processing software installed on the server, for example, the stereo matching technique and 
software, on a server at step S806 (S807). 

[0063] Moreover, according to authentication of a password etc., step S808 is a step which performs 
processing / transfer processing of the saved three-dimension data, and is set between a personal digital 
assistant with a camera, and a server. For example, mutual recognition processing by the common key 
authentication method or the public key authentication method is performed, the check of each communications 
partner is performed, access to the data stored in the server is permitted, and processing / transfer processing 
of the saved three-dimension data is performed. In addition, it is necessary for the communication terminal in 
this case to have the possible cipher-processing section of mutual recognition processing. By constituting the 
possible cipher-processing section of mutual recognition processing in a personal digital assistant with a 
camera, it is good also as a configuration which performs authentication processing, performs authentication 
processing among PCs with a camera and the servers which have the possible cipher-processing section of 
mutual recognition processing good also as a configuration and which performs data reception from a server, 
and performs data reception from a server. 
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[006,4] Drawing 9 is drawing having shown the three-dimension measurement technique, and the above- 
mentioned example and above-mentioned experimental result by the system. Drawin g 9 (a) and (b) are the 
images observed with the personal digital assistant 1 (criteria camera) with a camera, and the personal digital 
assistant 2 (detection camera) with a camera, respectively. In order to transmit image data to a high speed, the 
stereo image was saved in the jpg (j peg) format. Using the image of two sheets sent to the server, the camera, 
and the plinth parameter, stereo matching was performed and the result (depth map of a face (x, y, depth)) of 
drawing 9 (c) was searched for. Furthermore, conversion of a up to [ xyz space ] can be performed and it can 
ask for the three-dimension data (x y, z) of a face. Drawing 9 (d) and (e) are the face images when seeing the 
three-dimension configuration with a certain view. 

[0065] The example of a configuration of the personal digital assistant with a camera which contained the plinth 
in drawing 10 is shown. Drawin g 10 (a) is the example of a configuration in which the plinth 1003 was contained 
by the lower part of the personal digital assistant 1001 which has the camera section 1002 possible [ a cash 
drawer ], and drawing 10 (b) is the example of a configuration in which the plinth 1006 was contained possible 
[ folding ] by the flank of a personal digital assistant 1004 which has the camera section 1005. 
[0066] drawin g 10 (a) and (b) — each example of use is shown in drawin g 10 (c) and (d). Drawin g 10 (c) pulls out 
the plinth contained by the lower part of a personal digital assistant possible [ a cash drawer ], and is 
predetermined distance and a configuration which is made to estrange and sets other cameras. It has the slot 
or connection terminal for positioning of a camera in the plinth, and has in it the configuration as which the 
relative position of two cameras is determined at fixed distance and a fixed include angle. 
[0067] Moreover, the reference camera 1007 is connected to the connection terminal (not shown) of a plinth 
1008 by forming the signalling channel for performing actuation of two cameras to a plinth as processing which 
synchronized, for example, setting the reference camera 1007 to a plinth 1008. Moreover, processing signals, 
such as the shutter speed and the diaphragm which were explained by above-mentioned drawin g 4 . and a 
stroboscope, are also outputted and inputted between two cameras through the signalling channel in a plinth, 
and a connector terminal. 

[0068] Drawing 10 (d) pushes down the plinth contained by the flank of a personal digital assistant possible 
[ folding ], and is predetermined distance and a configuration which is made to estrange and sets other 
cameras. Like (c), it has the slot or connection terminal for positioning of a camera, the relative position of two 
cameras is determined at fixed distance and a fixed include angle, and it has the configuration which can output 
and input various signals in a plinth. 

[0069] Other examples of a configuration of a personal digital assistant with a camera are shown in drawin g 1 1 . 
Drawing 1 1 is the example which constituted the camera and the plinth in the personal computer (PC) of a 
mobile form. Drawing 1 1 (a) is the example of a configuration in which the plinth 1 103 was contained by the 
flank of the display section of PC1 101 which has the camera section 1 102 possible [ folding ]. and drawing 1 1 
(b) is the example of a configuration in which the plinth 1 106 was contained withdrawal by the keyboard section 
of PC1 104 which has the camera section 1 105. 

[0070] drawing 1 1 (a) and (b) — each example of use is shown in drawing 1 1 (c) and (d). Drawing 1 1 (c) pushes 
down the plinth contained by the flank of the display section of PC possible [ folding ], and is predetermined 
distance and a configuration which is made to estrange and sets the personal digital assistant as other 
cameras. It has the slot or connection terminal for positioning of a camera in the plinth, and has in it the 
configuration as which the relative position of two cameras is determined at fixed distance and a fixed include 
angle. 

[0071] Moreover, the personal digital assistant 1107 which is a reference camera is connected to the 
connection terminal (not shown) of a plinth 1 108 by forming the signalling channel for performing actuation of 
two cameras to a plinth as processing which synchronized, for example, setting a personal digital assistant 1 107 
to a plinth 1 108. Moreover, processing signals, such as the shutter speed and the diaphragm which were 
explained by above-mentioned drawin g 4 , and a stroboscope, are also outputted and inputted between two 
cameras through the signalling channel in a plinth, and a connector terminal. 

[0072] Drawing 1 1 (d) pulls out the plinth contained by the flank of the keyboard section of PC withdrawal, and 
is predetermined distance and a configuration which is made to estrange and sets other cameras. Like (c), it 
has the slot or connection terminal for positioning of a camera, the relative position of two cameras is 
determined at fixed distance and a fixed include angle, and it has the configuration which can output and input 
various signals in a plinth. 

[0073] The three-dimension image generation processing flow in the configuration of drawing 10 and the 
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configuration of drawing 1 1 is shown in drawing 12 . Drawing 12 (a) is a three-dimension image generation 
processing flow in the configuration of drawin g 10 , and drawin g 12 (b) is a three-dimension image generation 
processing flow in the configuration of drawing 1 1 . 

[0074] It explains from the processing flow of drawing 12 (a). First, 3D measurement mode in a personal digital 
assistant is set at step SI 201. 3D measurement mode is the photography mode in which have equipment which 
measures an illuminance by diaphragm as fixed shutter speed, using flash plate light, and a diaphragm is 
automatically acjjusted with fixed shutter speed, as drawin g 4 explained. At step SI 202, the camera of the 
personal digital assistant with which a plinth (attachment) is used is used as a criteria camera, and others are 
set up as a reference camera. At step SI 203, the photography method of a criteria camera is set up, it lets a 
plinth (attachment) pass and the photography method and parameter of a reference camera are set up. In 
addition, the parameter relevant to a photography method means shutter speed, a diaphragm, synchronous 
photography timing, etc. In step SI 204, synchronous photography by two sets of personal digital assistants is 
performed. At step SI 205, the photoed image, the interior parameter of a camera, and the physical relationship 
parameter decided by the plinth are transmitted to a device with 3D processing facility, and 3D configuration 
information is searched for. Specifically, you may transmit or process with a personal digital assistant with a 
count function through a wireless network etc. 

[0075] The processing flow of drawing 12 (b) is explained. Processing of drawing 12 (b) is a processing flow on 
condition of the configuration in which the three-dimension image generation processing program was stored in 
PC in the configuration which has PC explained by drawin g 1 1 . First, in step SI 21 1, 3D measurement mode on 
a personal computer (software) is performed. At step SI 21 2, it lets the cable inside a plinth (attachment) pass, 
and 3D measurement mode, a photography method, a parameter, etc. are transmitted and set to a personal 
digital assistant from a personal computer. In step SI 21 3, the camera on a personal computer with a camera 
and a personal digital assistant performs synchronous photography, and the photography image on a personal 
digital assistant, a camera parameter, etc. are transmitted to a personal computer through an attachment. 
Moreover, since the Ichinoseki charge between two cameras is decided by the plinth (attachment), the 
parameter is set to a personal computer side. At step SI 214, 3D configuration is measured and displayed with 
3D measurement software on a personal computer. 

[0076] In addition, although each of a personal digital assistant camera has explained the configuration which 
performs data transmission to a server in the above-mentioned example, it is good also as a configuration which 
transmits two or more images photoed with two or more cameras to one personal digital assistant through a 
plinth, and transmits all photography images to a server from the one terminal. 

[0077] As mentioned above, it has explained in detail about this invention, referring to a specific example. 
However, it is obvious that this contractor can accomplish correction and substitution of this example in the 
range which does not deviate from the summary of this invention. That is, with the gestalt of instantiation, this 
invention has been indicated and it should not be interpreted restrictively. In order to judge the summary of this 
invention, the column of the claim indicated at the beginning should be taken into consideration. 
[0078] 

[Effect of the Invention] As mentioned above, as explained, according to the three-dimension image generative 
system, three-dimension image generation method, and three-dimension information service system of this 
invention, following various effectiveness is brought about. (1) Set up fixed shutter speed and a fixed diaphragm, 
it has equipment which performs photography using flash plate light, or measures an illuminance, and it becomes 
possible to photo the image of the brightness value stabilized also out in the fields with shutter speed or 3D 
photography mode in which a diaphragm is adjusted automatically. (2) When two or more set (two or more sets) 
use of the personal digital assistant with a camera is carried out, and the plinth which installs those personal 
digital assistants with a camera is prepared beforehand and carries out a calibration, a high precision three- 
dimension shape measurement becomes possible. (3) It becomes sharable [ a three-dimension image ] to 
transmit the measured three-dimension data or the three-dimension configuration data saved to the server to 
other terminals using a personal digital assistant with a camera etc. (4) Since it corresponds to minute change 
of the physical relationship between cameras, the conventional technique is improved and high precision three- 
dimension measurement is attained with the stereo matching method using the corresponding-points heuristics 
on an epipolar line with a certain line breadth. (5) The three-dimension data utility of the three-dimension 
configuration data saved on the server which two or more users download the three-dimension image data 
stored in the server, or transmit to a friend through a server according to the enciphered password 
authentication becomes possible. (6) By considering as the configuration holding the table which matched the 
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identifier and parameter of a plinth with the server, it becomes possible to choose a parameter from a personal 
digital assistant with a camera based on the discernment data transmitted with a photography image, and to 
perform an exact three-dimension shape measurement and three-dimension image generation processing by 
the server side. 



[Translation done.] 
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* NOTICES * 

JPO and NCI PI are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 
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